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Abstract 

DoD  acquisition  is  an  extremely  complex  system,  comprised  of  myriad  stakeholders, 
processes,  people,  activities,  and  organizations  in  an  effort  to  provide  the  most  useful 
capabilities  to  warfighters  at  the  best  possible  value  to  the  government.  This  effort  is  being 
accomplished  by  acquisition  analysts  who  despite  years  of  experience  are  encumbered  by 
mountains  of  available  data.  To  assist  the  analyst,  we  consider  that  the  cognitive  interface 
between  decision-makers  and  a  complex  system  may  be  expressed  in  a  range  of  terms  or 
“features,”  i.e.,  specific  vocabulary  to  describe  attributes.  This  offers  the  opportunity  to  more 
easily  compare  two  competing  technologies,  which,  in  turn,  may  be  compared  to  the  Navy 
warfighter  requirements.  This  effort  can  allow  decision-makers  to  become  aware  of  what 
programs,  systems,  and  specific  features  are  available  for  acquisition  and  how  well  they 
match  warfighter’s  needs  and  requirements  with  greater  effect  and  immediacy — possibly  in 
real-time.  We  present  a  data-driven  automation  method,  namely,  Lexical  Link  Analysis 
(LLA),  to  facilitate  and  automate  acquisition  system  self-awareness. 

Introduction 

DoD  acquisition  is  an  extremely  complex  system,  comprised  of  myriad  stakeholders, 
processes,  people,  activities,  and  organizations  in  an  effort  to  provide  the  most  useful 
capabilities  to  warfighters  at  the  best  possible  value  to  the  government.  According  to  the 
Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  for  Joint  Capabilities  Integration  and 
Development  System  (JCIDS)  (J-8  CJCSI 31 70.01  G)  (JCIDS,  2009),  there  are  three  key 
processes  in  the  DoD  that  must  work  in  concert  to  deliver  the  capabilities  required  by  the 
warfighter:  the  requirements  process;  the  acquisition  process;  and  the  Planning, 
Programming,  Budget,  and  Execution  (PPBE)  process.  In  particular,  the  requirements 
process  is  implemented  in  a  process  called  Joint  Capabilities  Integration  and  Development 
System  (JCIDS),  as  shown  in  Figure  1.  JCIDS  plays  a  key  role  in  identifying  the  capabilities 
required  by  the  warfighters  to  support  the  National  Defense  Strategy,  the  National  Military 
Strategy,  and  the  National  Strategy  for  Homeland  Defense.  The  Defense  Acquisition  System 
(DAS)  looks  on  enterprise  asset  acquisition  based  on  JCIDS  requirements,  and  PPBE  is 
focused  on  the  management  of  financial  resources  in  accomplishing  enterprise  asset 
creation,  sustainment  and  reuse.  The  leadership  and  decision-makers  constantly  contend 
with  two  major  questions: 

1 .  Are  we  responding  to  strategic  guidance  and  joint  capability  needs? 

1 .  Are  we  getting  the  best  value  for  taxpayers? 

As  shown  in  Figure  1,  JCIDS  alone  produces  a  large  amount  of  detailed  documents 
(e.g.,  Initial  Capabilities  Document  (ICD),  Formal  Capability  Development  Document  (CDD), 
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for  material  solutions  or  doctrine,  organization,  training,  materiel,  leadership  and  education, 
personnel,  or  facilities  (DOTMLPF),  Change  Recommendations  (DCR)  for  non-material 
solutions,  and  Capability  Production  Document  (CPD)).  Each  involves  diversified 
stakeholders  such  as  sponsors,  program  managers,  developers,  the  Joint  Requirements 
Oversight  Council  (JROC)  and  the  Milestone  Decision  Authority  (MDA). 


-OgAt  Jags  lAl  v\ 


Prod  & 
Deployment 


□ 


Sponsor  Activity 


1=  JCIDS  Document 


OA 


=  Acquisition  decision 


Figure  1.  JCIDS  Process  and  Acquisition  Decisions 

(JCIDS,  2009) 

Warfighters’  requirements  are  documented  in  Universal  Joint  Task  List  (UJTLs)  or 
Joint  Capability  Areas  (JCAs),  which  are  collections  of  required  capabilities  functionally 
grouped  to  support  mission  analysis,  capability  analysis,  strategy  development,  investment 
decision-making,  capability  portfolio  management,  and  capabilities-based  force 
development  and  operational  planning. 

Multiple  Portfolio  Views: 

•  Systems  vs.  Capabilities 

•  Investment  vs.  Capabilities 

•  System  Context 

•  Highly  dependent  programs 
(Joint  Enablers) 

•  Procurement  Optimization 

•  S&T  vs.  future  needs 

•  Sustainment  Efficiency 

•  Market  Value 


Figure  2.  Portfolio  Analytic  Capability 

(Appleton,  2009) 

In  summary,  the  major  challenges  in  the  current  process  can  be  summarized  as 

follows: 


2.  To  make  optimal  investment  decisions,  acquisition  managers  must  analyze  a 
full  spectrum  of  data,  including  data  that  encompasses  capability 
requirements,  planning,  development,  integration,  testing,  architecture, 
standards,  cost  and  schedules.  This  can  be  a  daunting,  if  not  impossible, 
task. 

2.  The  pace  of  technology  change  also  requires  agile  decision-making 
and  challenges  program  management  to  maintain  constant 
awareness  of  what  is  available  for  acquisition. 

3.  When  considering  an  overall  demand  and  supply  in  the  trade  space 
management  of  the  Department  of  Defense,  as  shown  in  Figure  2, 
decision-makers  require  advanced  portfolio  analytic  capability  that  can 
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intercept  all  three  business  processes  of  requirements,  acquisition 
and  PPBE  under  the  DoD  warfighting  strategic  guidance  in  the 
contexts  of  many  factors,  such  as  systems  versus  capabilities, 
investment  versus  capabilities,  highly  dependent  programs,  etc.,  in 
order  to  maximize  Return  of  Management  (ROM)  and  Yield  on  Cost 
(YOC)  (Appleton,  2009). 

4.  The  information  produced  in  the  process  is  too  voluminous  and 
unformatted  to  lend  itself  to  analysis  on  a  large  scale.  Decision¬ 
makers  require  large-scale  automation  and  discovery  tools  that  can 
speed  up  the  analysis  quickly  in  response  to  the  pace  of  technology 
change,  therefore  adapting  DoD  program  development  and 
associated  funding  mechanisms  in  an  agile  manner.  The  decision¬ 
makers  also  require  a  much  more  fine-grained  level  of  analysis  for 
program-to-program  and  program-to-program  elements  analysis  using 
the  unstructured  documents  directly.  This  is  a  big  leap  that  is  not 
provided  by  the  current  analysis  capabilities. 

One  method  to  reduce  unknown  performance  measures  is  through  participation  in 
annual  large-scale  field  experimentation  exercises  as  part  of  the  Research,  Development, 
Test  &  Evaluation  (RDT&E).  These  experiments  can  provide  close  interaction  among  users, 
developers,  the  test  community,  and  decision-makers.  At  Distributed  Information  Systems 
Experimentation  (DISE)  laboratory  at  NPS,  we  collect  and  analyze  data,  help  the  Navy  learn 
and  manage  information  and  knowledge  resulting  from  large-scale  annual  experimentation 
(e.g.,  Trident  Warrior  and  Empire  Challenge).  We  believe  this  experiential  data,  together 
with  Lexical  Link  Analysis  methods,  will  produce  deepened  awareness  of  current  program 
effectiveness  for  acquisition  decision-makers. 

Methods 

Program  Self-awareness 

Here  we  consider  that  the  cognitive  interface  between  decision-makers  and  a 
complex  system  may  be  expressed  in  a  range  of  terms  or  “features,”  i.e.,  specific  vocabulary 
or  lexicon,  to  describe  attributes  and  the  surrounding  environment  of  a  system.  This 
process  is  similar  or  can  be  modeled  using  human  cognitive  processes,  where  the  simplest 
form  of  such  a  model  is  relationships  between  noun/verb.  In  math,  the  model  becomes 
variable/function;  in  engineering  it  becomes  operand/operator;  in  information  technology,  it 
becomes  data/process  or  description/procedure.  We  have  borrowed  from  notions  of 
“awareness,”  and  implement  the  term  self-awareness  of  a  complex  system  as  the  collective 
and  integrated  understanding  of  system  features.  A  related  term,  “situational  awareness”  is 
used  in  military  operations  and  carries  with  it  a  sense  of  immediacy  and  cognitive 
understanding  of  the  warfighting  situation.  Here,  system  self-awareness,  or  program 
awareness  (Gallup,  MacKinnon,  Zhao,  Robey  &  Odel,  2009),  allows  decision-makers  to  be 
aware  of  what  systems,  programs,  and  products  are  available  for  acquisition,  how  they 
match  warfighters’  needs  and  requirements,  recognize  relationships  among  them,  improve 
efficiency  of  available  collaboration,  reduce  duplication  of  effort,  and  re-use  components  to 
support  cost  effective  management — with  greater  immediacy,  possibly  in  real-time. 

Through  our  research,  we  present  a  data-driven  automation  method,  namely,  a 
Lexical  Link  Analysis  (LLA)  for  program  self-awareness.  This  methodology  is  demonstrated 
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by  extracting  realistic  sample  data  related  to  systems  and  programs  included  in 
experimentation  programs,  Urgent  Needs  Statements  (UNS),  and  CENTCOM/NAVCENT 
warfighting  gap/priority  lists,  a  large-scale  data  set  from  OSD  with  regards  to  Major  Defense 
Acquisition  Programs  (DMAP)  and  Acquisition  Category  II  (ACATII)  weapon  systems  and 
their  RDT&E  documentations. 

Lexical  Link  Analysis  (LLA) 

Data  mining  includes  analytic  tools  that  may  be  applied  to  both  structured  and 
unstructured  data  to  confirm  previously  determined  patterns,  or  to  discover  new  patterns 
that  are  yet  unknown.  Text  mining  is  the  application  of  data  mining  to  unstructured  or  less 
structured  text  files.  Text  mining  represents  an  emerging  field  with  a  wide  range  of  software 
implementing  innovative  visualization  and  navigation  techniques.  These  techniques 
graphically  represent  networks  of  documentation  that  are  related  conceptually.  Visualization 
of  relationships  enables  concept  discovery,  automated  classification,  and  understandable 
categorization  of  unstructured  documents. 

Lexical  Analysis  (LA,  2010)  is  a  form  of  text  mining  in  which  word  meanings  are 
developed  from  the  context  from  which  they  are  derived.  Lexical  Analysis  (LA)  can  also  be 
used  in  a  learning  mode,  where  such  words  and  context  associations  are  initially  unknown 
and  are  constantly  being  “learned,”  updated,  and  improved  as  more  data  become  available. 
Link  analysis,  a  subset  of  network  analysis  that  explores  associations  between  objects, 
reveals  the  crucial  relationships  between  objects  when  collected  data  may  not  be  complete. 
Lexical  Link  Analysis  (LLA)  is  an  extended  lexical  analysis  and  link  analysis  enabled  in  a 
learning  mode. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


269 


Figure  3.  A  Word  Hub  Showing  the  Detail  on  the  Linkage  in  Figure  3 

This  approach  clusters  words  and  then  correlates  words  with  their  textual  contexts 
(co-occurrence),  and  produces  a  data-driven  and  dynamic  word  network.  This  approach  is 
related  to  a  number  of  extant  tools  for  text  mining,  including  Latent  Semantic  Analysis  (LSA) 
(Dumais  et  al.,  1998),  advanced  search  engine  (Foltz,  2002),  key  word  analysis  and  tagging 
technology  (Gerber,  2005),  and  intelligence  analysis  ontology  for  cognitive  assistants 
(Tecuci  et  al.,  2007).  What  results  from  this  process  is  a  learning  model — like  an 
ethnographic  code  book  (Schensul,  Schensul  &  LeCompte,  1999) — containing  descriptions 
of  both  patterns  and  anomalies,  generated  using  encountered  terms.  As  an  example  shown 
in  Figures  3  and  4,  we  applied  our  approach  to  Maritime  Domain  Awareness  (MDA) 
technologies  that  were  evaluated  in  Trident  Warrior  08.  Figure  3  shows  a  visualization  of 
LLA  with  connected  keywords  or  concepts  extracted  from  the  documents  of  MDA 
technologies.  Words  are  linked  as  word  pairs  that  appear  next  to  each  other  in  the  original 
documents.  Different  colors  indicate  different  clusters  of  centralization  among  word  groups. 
They  are  produced  using  a  link  analysis  method,  a  social  network  grouping  method  (Girvan 
&  Newman,  2001 ):  words  are  connected  as  shown  in  one  color  as  if  they  are  in  a  social 
community.  A  “hub”  is  a  word  centered  with  a  list  of  other  words  (“fan-out”  words)  centered 
around  other  words.  For  instance,  in  Figure  4,  the  word  “behavior”  is  centered  with 
“suspicious,  bad,  dangerous,  abnormal,  usual,  and  anomalous,”  etc.,  showing  the  ways  to 
describe  “behavior”  in  the  MDA  area. 

Figures  5  and  6  show  a  visualization  of  lexical  links  between  Systems  1  and  2.  Each 
node  is  a  feature,  or  word  hub;  each  color  refers  to  the  collection  of  lexicon  (features)  to 
describe  a  system,  the  overlapping  area  nodes  refer  to  lexical  links  between  systems.  The 
nodes  toward  the  two  ends  of  the  links  represent  the  unique  features  related  to  each 
system. 


DANGEROUS 


Figure  4.  A  Word  Hub  Showing  the  Detail  on  the  Linkage  in  Figure  3 
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Unique  Features 
System  1 


Lexical  Links  between 
System  1  and  2 


Unique  Features  of 
System  2 


Figure  5.  Visualization  of  Lexical  Links 
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TW10  Technologies 


Overlapping  terms  or  lexical  links  are  inside  the 
box  in  the  middle  (not  the  real  data) 


Figure  6.  Overlapping  Terms  or  Lexical  Links,  Shown  in  the  Middle  of  Two  Word 

Networks  as  the  Result  of  the  LLA  Analysis 


In  summary,  LLA  provides  a  methodology  and  tools  to  address  the  following  specific 
areas  that  can  impact  acquisition  decision-making: 


■  LLA  provides  a  metric  to  link  warfighters’  needs  with  the  capabilities  by  directly 
comparing  the  documents  that  resulted  from  the  business  process — for  example, 
linking  “programs,”  specifically  MDAPs,  to  operational  capabilities.  The  number 
of  lexical  links,  extracted  to  reflect  the  meaning  of  the  documents  between  two 
systems  or  programs,  can  be  a  measure  of  consensus  or  synergy  between  the 


TtsJPO I  ACQUISITION  RESEARCH  PROGRAM 

?/fg7  GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 
NAVAL  POSTGRADUATE  SCHOOL 


271 


two.  This  compelling  perspective  is  central  to  the  notion  of  portfolio 
management,  for  example,  to  answer  the  questions:  What  are  the  programs 
(e.g.,  MDAPs)  related  to  a  given  capability?  What  are  the  gaps  of  warfighter 
requirements  not  addressed  by  current  programs?  Currently,  human  analysts 
are  responsible  to  answer  these  questions  manually.  Automation  is  needed  to 
facilitate  human  analysis  and  to  process  large  volumes  of  data  quickly. 

■  LLA  visualization  is  also  important  for  acquisition  decision-making.  Producing  a 
picture  illustrating  where  the  needs  are  met  and  where  the  overlapping  efforts 
and  gaps  are  will  allow  decision-makers  to  become  aware  of  the  overall  situation, 
thus  allowing  them  to  see  trends  in  a  larger,  broader  scale  and  in  a  longer 
timeframe.  For  example,  combining  the  analyses  of  the  Army,  Navy,  and  Air 
Force  from  RDT&E  and  procurement  documents  might  show  the  linkages  within 
and  among  programs,  as  they  mature  from  development  to  production.  Modified 
programs  can  be  illustrated  to  show  the  trend  toward  (or  deviation  away  from) 
warfighters’  needs  during  the  program’s  life  span.  One  may  also  visually  see  the 
resource  sharing  (or  wasting)  practices  and  note  opportunities  for  growth  when 
all  the  data  can  be  summarized  in  a  discernable  picture. 

■  LLA  discovers  latent,  implicit,  or  second-order  relationships  by  examining  the 
detailed  budget  justification  documents.  In  general,  programs  retain  their 
identities  from  development  to  production,  yet  may  change  their  names  or  be  re¬ 
designated,  resulting  from  a  milestone  decision  or  other  action.  The  "New  Attack 
Sub"  or  "NSSN"  during  development,  for  instance,  was  referred  to  as  the 
"Virginia  Class  Sub"  in  production.  The  "Joint  Strike  Fighter"  and  "F-35"  are  also 
synonymous.  The  official  "decoder"  for  these  transformations  is  the  DAMIR 
system.  We  note  that  the  mapping  of  MDAPs  to  their  predecessors,  successors, 
constituents,  or  dependent  partners  is  non-trivial  and  is,  in  fact,  one  of  the 
fundamental  challenges  for  acquisition  analysts. 

■  LLA  could  affect  the  fundamentals  of  acquisition  processes  through  automation 
and  discovery.  In  the  defense  acquisition  community,  decision-makers  are 
interested  in  determining  the  costs  of  these  programs  relative  to  their  predicted 
baselines  (e.g.,  Milestone  B  or  C).  They  must  also  determine  why  costs  change 
over  time.  Historically,  acquisition  researchers  only  considered  endogenous 
factors  (e.g.,  poor  program  management  skills)  as  drivers  of  cost  changes.  The 
notion  of  interdependence  as  a  potential  driver  of  cost  may  be  determined  by 
LLA.  It  may  also  help  determine  whether  this  interdependence  among  programs 
may  be  manifested  in  the  sharing  of  resources  among  programs,  as  described  by 
the  budget  artifacts.  Budget  artifact  data  are  voluminous,  and  unstructured, 
which  make  empirical  analysis  extremely  difficult — if  not  humanly  impractical. 
Previous  research  has  been  done  in  this  area  using  manually  identified  program 
interdependencies  (M.  Brown,  personal  communication,  2010)  and  has  made 
great  progress  in  establishing  that  interdependence  exists  and  how  they  might  be 
correlated  with  the  program  costs.  LLA  could  automate  this  process  of 
identifying  interdependencies  and,  thus,  reveal  aspects  of  interdependence  that 
would  otherwise  remain  obscure. 
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LLA  Processes 


The  LLA  Analysis 

We  began  at  the  Naval  Postgraduate  School  (NPS)  by  using  Collaborative  Learning 
Agents  (CLA)  (Ql,  2009)  and  expanded  to  other  tools,  including  AutoMap  (AutoMap,  2009) 
for  improved  visualizations.  Results  from  these  efforts  arose  from  leveraging  intelligent 
agent  technology  via  an  educational  license  with  Quantum  Intelligence,  Inc.  CLA  is  a 
computer-based  learning  agent  or  agent  collaboration,  capable  of  ingesting  and  processing 
data  sources.  Each  CLA  is  capable  of  revealing  patterns  that  occur  frequently  and 
anomalies  that  occur  rarely.  Anomalies  that  might  be  interesting  are  thus  revealed  so  that 
human  analysts  are  alerted  and  can  further  investigate  them.  The  CLA  is  able  to  separate 
the  patterns  from  anomalies  using  the  “patterns  and  anomalies  separation”  algorithm  in 
each  CLA  to  select  feature-like  word  pairs  for  the  LLA  method. 

The  following  are  the  steps  for  the  LLA  analysis: 

3.  Read  two  documents  into  the  CLA  (e.g.,  Urgent  Needs  Statement  (UNS))  and 
a  targeted  technology  document  set  (e.g.,  Trident  Warrior  2010  (TW10). 

5.  Select  feature-like  word  pairs  based  on  clusters  using  the  CLA 
anomaly  search  method  (Zhao  &  Zhou,  2008). 

6.  Apply  social  network  algorithm  to  group  the  word  pairs  into  word 
categories. 

7.  Apply  AutoMap  to  visualize  the  associations  of  the  requirement 
document  set  (UNS)  and  targeted  technologies  (TW10)  document 
sets,  as  shown  in  Figures  5  and  6. 

8.  Generate  lexical  link  matrices  used  for  further  analyses,  as  shown  in 
Figures  8,  9,  and  10. 

When  mining  text  data  or  performing  lexical  analysis,  we  also  apply  entity  extraction, 
known  as  Named  Entity  Recognition  (NER),  (NER,  2010;  Nadeau,  Turney  &  Matwin,  2006), 
which  recognizes  named  entities  such  as  persons,  organizations,  locations,  expressions  of 
times,  quantities,  monetary  values  and  percentages  in  context.  The  extracted  entities  could 
also  be  examined  separately.  Excluding  these  modifiers  from  the  terms  resulting  from 
Lexical  Link  Analysis  (LLA)  can  provide  an  improved  comparison  by  focusing  on  term 
semantics. 

In  some  applications,  differentiating  nouns  from  verbs  and  adjectives,  or  having  the 
ability  to  parse  the  syntax  into  nouns,  verbs,  subjects,  and  objects,  could  be  helpful  to 
acquisition  managers  to  develop  understanding.  We  also  use  a  Part-of-Speech 
(POS)  tagger  as  pre-  or  post-processing  filters  for  this  purpose.  A  POS  tagger  is  a  piece  of 
software  that  reads  text  in  some  language  and  assigns  parts  of  speech  to  each  word,  such 
as  a  noun,  verb,  adjective,  etc.  We  have  chosen  the  Stanford  Natural  Language  Processing 
(NLP)  tool  (Toutanova,  Klein,  Manning  &  Singer,  2003;  Stanford  NLP,  2009)  to  perform  this 
task.  The  POS  taggers  are  usually  language  dependent.  Our  method  is  statistically  based 
and  can,  therefore,  employ  NER  and  POS  as  pre-  or  post-processing  filters. 
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Data  Sets 


We  report  a  case  study  using  LLA  comparing  US  Navy  Urgent  Need  Statements 
(UNS)  with  Trident  Warrior  10  Technologies.  The  goal  is  to  compare  the  two  respective 
data  sets,  the  first  one  is  an  Excel  file  (UNS.xls)  representing  Urgent  Need  Statements 
collected  from  C4I  users.  Each  urgent  need  is  listed  as  a  statement.  The  UNS.xls  is 
classified;  therefore,  details  of  this  document  set  are  not  reported  in  this  paper.  The  second 
data  set  is  called  “Focus  Area  Assignment  TW  10.xls,”  also  in  an  Excel  format.  It  includes 
information  from  each  selected  technology  in  Trident  Warrior  10. 

Trident  Warrior  (TW)  is  an  annual  Navy  FORCEnet  operational  experiment.  At  the 
Distributed  Information  Systems  Experimentation  (DISE)  laboratory  at  NPS,  we  collect  and 
analyze  data  from  this  and  other  experimentation  venues  to  help  the  Navy  learn  and 
manage  information  and  knowledge  resulting  from  large  field  experiments  such  as  Trident 
Warrior  to  provide  a  basis  for  DoD  acquisition  of  systems  and  technologies.  The  technology 
information  includes  each  technology’s  objective(s)  for  the  experimentation,  including 
Concept  of  Operations  (e.g.,  how  a  warfighter  will  utilize  it),  and  what  each  technology 
provider  intends  to  learn  from  the  experimentation  (e.g.,  decrease  timeline,  standardized 
process,  and/or  reduced  workload,  etc.).  TW  data  also  includes  decisions  that  may  affect 
experimentation  findings. 

Result  Presentation  and  Visualization  Tools 

Figure  7  illustrates  a  result  summary  revealing  terms  or  word  pairs  combined  into 
word  categories,  displayed  in  a  radial  graph.  The  categories  with  radius  =  2  represent 
overlapping  word  categories  that  are  found  in  both  requirements  (UNS)  and  technologies 
(TW10).  The  categories  with  radius  =  1  indicate  where  gaps  exist,  i.e.,  terms  that  show  in 
the  UNS  but  not  in  the  TW10  technologies  or  vice  versa.  We  determine  that  there  is 
between  a  60%  and  70%  match  overlap  of  technology  correlations  between  UNS  and  TW 
10  technologies.  For  example,  42  of  67  (62%)  of  the  UNS  word  categories  matched  (were 
served  by)  with  TW10  technologies. 

In  addition,  word  network  views  of  lexical  links  are  produced  using  a  network  tool, 
AutoMap.  We  also  developed  several  outputs  to  view  the  detailed  LLA  analysis  results  as 
shown  in  Figures  8,  9,  and  10.  Figure  8  shows  an  Excel  document  output,  including  a  few 
columns  of  information  as  follows: 

■  Terms:  Matching  terms  or  word  categories  discovered  automatically  via  the  LLA 
method. 

■  UNS:  Values  can  be  0,  1, 2,  specifically: 

o  0:  terms  not  found  in  UNS, 

o  1:  terms  only  found  in  UNS,  and 

o  2:  terms  found  in  both  UNS  and  TW10. 

■  UNS  IDS:  UNS  documents  in  which  the  terms  can  be  found. 

■  TW10:  Values  can  be  0,  1, 2. 

o  0:  terms  not  found  in  TW10, 

o  1:  terms  only  found  in  UNS,  and 
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o  2:  terms  found  in  both  UNS  and  TW10. 

■  TW10  IDS:  TW10  documents  in  which  the  terms  can  be  found. 

■  Tech  Features:  Terms  only  belong  to  TW10. 
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As  one  scrolls  down,  if  there  is  “0”  in  the  TW10  column,  then  it  indicates  a  gap 
area  forTWIO.  Similarly,  in  scrolling  further,  if  there  is  a  “0”  in  the  UNS  column, 
then  this  indicates  a  gap  in  UNS. 


Overlapping  categories 


Categories  found  automatically 


42  of  67  (62%)  of  UNS 
are  matched  in  TW10 


Figure  7.  Resulting  View  and  Visualization  Illustrating  “Overlapping”  and  “Gap” 
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Figure  8.  The  Spreadsheet  View  of  the  LLA  Analysis  with  “Matched”  Terms  and  “Gap” 

Terms 


Figure  9.  The  Matrix  View  of  the  LLA  Analysis 

Figure  9  shows  a  matrix  view  of  UNS  to  TW  10  technologies.  Where  numbers  are 
seen  indicates  a  numerical  reference  to  the  number  of  the  "concepts"  (terms  or  word 
categories)  included  between  UNS  and  technologies  that  are  being  satisfied.  Usually,  there 
are  multiple  concepts  within  a  UNS  statement  and  a  tech  description.  Each  number  is  also  a 
hyperlink  back  to  the  original  document  in  a  server  where  it  is  stored,  e.g.,  the  server  in  the 
NPS  Secure  Technology  Battle  Lab  (STBL)  for  classified  documents. 

These  results  can  be  increasingly  focused  as  the  Intelligent  Agent  (IA)  becomes 
“tuned,”  or  learns  what  it  is  that  the  researcher  is  attempting  to  understand.  This  effort  can 
then  become  increasingly  automated. 
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Figure  10.  Frequency  Count  and  Document  References 

Figure  10  shows  a  summary  spreadsheet  listing  the  terms  and  number  of  files  in 
which  the  terms  appear.  This  output  can  be  used  to  discover  concepts  (terms)  that  are 
cross-validated  by  at  least  two  documents  in  a  document  set.  The  terms  are  sorted  by  the 
number  of  "fan  out"  (the  words  connected  to  a  word  hub),  showing  the  critical  concepts 
being  addressed  across  multiple  documents.  The  top  few  sorted  word  groups,  e.g.,  “data” 
and  “information”  in  this  case,  are  the  key  requirements  that  result  in  substantial  consensus 
across  different  levels  of  requirement  generation  mechanisms — for  example,  Joint 
Integrating  Concept  (JIC),  Joint  Capability  Areas  (JCA),  the  Universal  Joint  Task  List  (UJTL), 
and  user  communities  such  as  US  Northern  Command,  US  Pacific  Command,  and  sponsors 
that  are  interested  in  Interagency  Investment  Strategies  (lISs). 


Validity 

Several  methods  are  being  investigated  to  validate  LLA  methods.  Currently,  we 
have  shown  these  proof-of-concept  results  to  Subject-matter  Experts  (SME)  from  various 
organizations  (e.g.,  Joint  Force  Development  and  Integration,  the  J-7  Staff)  for  evaluation 
and  comment.  One  MDA  expert  has  commented  on  the  summary  spreadsheet  by  saying,  “it 
is  very  useful,  particularly  the  frequency  count  and  the  documented  reference.”  Other  SMEs 
comment  that  “LLA  has  great  potential  to  help  us  link  the  UNS  with  the  technology  and 
further  fill  in  the  gaps  that  are  out  there.”  “This  would  be  highly  useful  and  has  great  potential 
to  help  us  in  the  larger  N9/Sea  Trial  construct  and  spoke  further  of  the  possibility  of  using 
LLA  at  the  Joint  Warfighter  Challenges  level.”  We  will  consider  quantitative  content 
validation  methods  between  SMEs  and  LLA,  such  as  correlation  and  inter-rater  reliability 
scores  (Cohen's  Kappa;  Kerlinger  &  Lee,  1992),  as  well  as  large-scale  correlation 
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calculation  used  in  sections  below. 


Towards  a  Large-Scale  Example  of  Program  Self-Awareness 

We  have  worked  with  OUSD(AT&L)/ARA/EI  on  the  broader  data  sets  and  a  large- 
scale  application  of  program  self-awareness  via  LLA. 

Data  Sets 
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Figure  11.  DoD  Budget  Documentation 


Figure  12.  Research,  Development  Test 
&  Evaluation  (RDT&E) 
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Exhibit  R-2,  RDT&E  Budget  'W!t\  Justification 

Narrative 
Justification 
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Navxtar  Global  Posiaoning  System  (GPS)  is  a  space-based  radio  positioning,  navigation,  and  time  (PNT)  distribution  system.  This  Program  Element  (PE)  funds  the 
Research  and  Development  (TLStD)  for  GPS  Cl  space  vehicles  (SV)  and  the  next  generation  Control  Segment  (OCX).  This  includes  but  is  not  limited  to.  advanced 
concept  development,  systems  engineering  and  analysis,  satellite  systems  development  the  study  of  augmentation  systems,  modernized  control  segment 
development,  user  equipment  interfaces,  training  simulators.  Integrated  Logistics  Support  dLS)  products,  and  developmental  test  resources. 


Funds  will  support  engineering  studies  and  analyses  architectural  engineering  studies,  trade  studies,  systems  engineering,  system  development,  test  and  evaluation 
efforts .  and  mission  operations  in  support  of  upgrades  and  product  improvements  for  military  and  civil  applications  necessary'  to  support  efforts  to  protect  U.S. 
military  and  allies*  use  of  GPS.  Additionally,  hinds  will  ensure  a  disciplined  Capability  Insertion  Program  plan  to  meet  Joint  Requirements  Oversight  Council 
(JROQ  approved  required  capabilities  Funds  util  support  science  and  techno  logs*  technology'  development  and  systems  development  to  meet  a  Block  approach 
<i  e  .  Block  m  A.  Block  IE  B.  etc  ). 


In  the  FY07  PB.  a  restructure  of  the  GPS  III  program  provided  funds  for  the  GPS  QI  SV  and  OCX  The  FYOS  PB  completes  the  GPS  III  restructure  Funding  for 
OCX  supports  an  additional  Prime  Contractor  to  support  OCX  concept  development  which  includes,  in  addition  to  GPS  m  capabilities,  the  ability  to  control 
modernized  signals. 


This  program  is  Budget  Activity  4  -  Advanced  Component  Development  and  Prototypes  because  it  is  in  Phase  A  (Concept  Development). 
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Figure  13.  Program  Element  RDT&E  Budget  Justification 


4.  We  have  obtained  program  element  (PE)  data,  which  are  used  for  DoD 

budget  justification  each  year,  as  shown  in  Figure  11.  One  PE  component  is 
Research,  Development,  Test  &  Evaluation,  which  is  the  budget  estimation, 
allocation  and  justification  used  for  programs  in  the  earlier  stages  of 
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development.  The  procurement  of  PE  components  is  the  counterpart  used  for 
mature  products.  RDT&E  books  are  obtained  from  the  Air  Force,  Army 
(http://asafm. army. mil/Document.aspx?OfficeCode=1 200)  and  Navy 
(http://www.finance.hq.navy.miI/fmb/1 1  pres/BOOKS.htm)  websites. 

9.  The  Weapon  Book  (Weapon,  2008),  which  summarizes  weapons  and 
their  basic  functions  and  missions,  combined  total  cost  from  RDT& 
and  procurement. 

10.  MMT  databases  contain  cost  and  schedule  information  for  each 
program.  They  consist  of  MDAPs  and  weapon  systems.  MMT 
databases  also  contain  various  program  interdependencies  identified 
by  human  analysts  that  can  be  used  for  validation.  MMT  databases 
also  contain  JCAs  and  UJTLs  mapped  to  programs  that  are 
handmade  by  human  experts. 

According  to  program  managers  Data  (1)  and  (2)  are  so  voluminous,  unformatted 
and  unstructured  that  traditional  analysis  methods  are  difficult  to  apply  on  this  scale; 
therefore,  they  are  the  major  focuses  of  the  analysis  for  LLA.  There  are  about  -500  PEs 
and  -80  weapon  systems  extracted  from  data  sets  (1)  and  (2),  with  a  total  size  about 
-200M.  Data  (3)  is  unstructured  and  various  previous  research  has  been  conducted  on  this 
data  and,  therefore,  can  be  used  to  validate  the  LLA  method  against  human  analyses. 

LLA  Analysis 

The  focus  of  this  paper  is  to  show  that  the  LLA  method  is  capable  of  improving 
system  self-awareness.  LLA  is  able  to  produce  this  by  providing  an  improved  methodology 
and  toolset  for  automation  and  discovery  of  patterns  and  anomalies  within  structured  and 
unstructured  data.  This  discovery  can  be  used  to  produce  graphics  illustrating  gaps  and 
overlaps  existing  between  systems  and  the  needs  of  the  DoD  by  basing  comparisons  on  the 
features  of  each  system.  This  methodology  can  have  the  effect  of  improved  savings  for  the 
DoD,  while  developing  high-value  products  that  meet  warfighters’  needs. 
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Figure  14.  An  Example  of  LLA  Matrices  of  Program  Elements  (PE)  against 

UJTLs 

First,  we  want  to  show  how  LLA  provides  a  new  metric  to  measure  how  warfighters’ 
needs  are  matched  with  resources  and  products  that  are  being  considered.  Figure  14 
shows  an  LLA  matrix  result  using  program  elements  as  columns  and  UJTLs  as  rows.  The 
number  in  each  cell  is  a  match  score  generated  from  the  LLA  method.  Next  to  the  score  are 
word  hubs  that  indicate  which  term  is  matched.  Sorting  this  matrix  according  to  the  matched 
scores  vertically  and  horizontally  answers  the  following  questions: 

■  Which  programs  (e.g.,  MDAPS)  are  related  to  a  given  capability?  Which  PEs  are 
related  to  a  given  capability? 

■  How  is  the  acquisition  process  responding  to  expressed  capability  needs?  How 
much  of  the  weapon  systems  acquisition  budget  is  being  allocated  to  any  given 
operational  need  (e.g.,  UJTL). 

Note  that  this  LLA  matrix  can  be  generated  for  any  pair  of  document  collections  that 
are  desired  for  comparison,  e.g.,  PEs  versus  UJTLs,  weapon  systems  versus  UJTLs  and 
weapon  systems  versus  weapon  systems.  When  applied  to  weapon  systems  (MDAPs) 
versus  UJTLs,  we  can  answer  the  following  question  by  sorting  the  LLA  matching  scores: 

■  Which  capability(ies)  does  any  given  MDAP  support?  How  much  does  the 
MDAP  contribute  to  this  capability? 

The  LLA  matrices  may  also  help  to  reconcile  the  gaps  between  the  final  products 
and  what  warfighters  need  after  the  long  process  of  design  and  development.  Furthermore, 
they  may  also  provide  new  prospective  for  portfolio  analysis.  A  conventional  treatment  of 
portfolio  analysis  is  that  it  is  typically  expressed  as  a  simple  correlation  between  an  MDAP 
and  a  capability.  This  simple  correlation  ignores  the  fact  that  no  individual  program  (system, 
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platform,  etc.)  can  contribute  to  any  capability  unless  other  programs/systems/capabilities 
are  in  place.  The  analogy  is  that  a  fighter  jet  is  useless  unless  it  has  all  the  supporting 
capabilities/infrastructure  (airfield,  ammo,  fuel,  personnel,  etc.),  and  complementary  systems 
(e.g.,  GPS,  C2,  satellite  imagery,  mission  planning,  etc.)  to  enable  it  to  operate  effectively. 
Considering  a  single  MDAP  in  terms  of  how  much  it  contributes  to  a  given  capability  without 
considering  its  linkages  to  other  systems/programs/capabilities  might  be  counterproductive, 
and  would  likely  drive  bad  decisions.  The  better  approach  is  to  consider  a  program  in  the 
context  of  its  interdependencies  with  respect  to  their  collective  contribution  to  a  specific 
capability.  The  interdependencies  should  be  identified  from  operational  needs,  engineering 
constructions  and  programmatic  budget  justifications.  Therefore,  the  combinations  of  the 
LLA  matrices — for  example,  PEs  versus  UJTLs,  weapon  systems  versus  UJTLs  and 
weapon  systems  versus  weapon  systems  may  also  help  to  redefine  portfolios  and  improve 
portfolio  management. 

Validity 

In  order  to  realize  the  potential  of  the  LLA  method,  an  important  first  step  is  to 
establish  the  validity  of  the  method  in  the  context  of  realistic  large-scale  data  sets.  For  that, 
we  used  the  matrix  generated  from  PEs  versus  PEs,  compared  with  what  human  analysts 
have  identified  previously.  As  shown  in  Figure  15,  in  each  program  element  artifact,  another 
program  element  might  be  referenced,  indicted  as  precedent  or  directionally  linked  program 
elements.  A  backward  link  is  usually  a  stronger  indicator  of  importance  of  a  PE  than  a 
forward  link.  This  is  similar  to  the  information  retrieval  or  page  ranking  in  a  search  engine 
(e.g.,  Google).  Here,  we  use  the  number  total  forward  and  backward  links  together, 
identified  by  human  analysts,  as  the  attributes  to  validate  the  LLA  method.  For  example, 
Figure  15,  PE  0604602F  references  PE  060501 1 F,  in  which  we  define  it  as  a  forward  link, 
for  PE  0604602F;  while  PE  060501 1 F  is  referenced  by  PE  0604602F,  which  we  define  as  a 
backward  link,  for  PE  060501 1 F.  As  shown  in  Figure  16,  the  top  yellow  row  contains  the 
total  number  of  unique  word  hubs  for  a  PE,  matched  with  all  PEs  other  than  itself;  and  the 
bottom  yellow  row  contains  the  total  number  of  forward  and  backward  links  for  the  same  PE. 
The  Pearson  correlation  of  the  two  rows  is  0.39,  with  a  p-value  <  0.0000001  (bi-directional  t- 
test  with  a  sample  size  N=461).  This  indicates  that  the  positive  correlation  between  the  LLA- 
identified  links  and  human-analyst-identified  links  is  statistically  significant  and,  therefore,  is 
a  validation  for  the  LLA  method. 
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Figure  16.  The  Correlation  Between  LLA  Word  Hubs  and  PE  Links  Identified  by  SME’s  is 

Statistically  Significant 
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Acquisition  Decision-making 


To  support  effective  decision-making,  we  need  to  form  a  full  understanding  of  a 
program  in  context;  we  need  to  understand  the  linkages  and  interdependencies  across  the 
operational,  constructive,  and  programmatic  domains. 

An  LLA  matrix  using  programs  such  as  weapon  systems  as  rows  as  well  as  columns 
is  shown  in  Figure  17.  The  lexical  links  output  from  this  view  show  the  relationships  among 
weapon  systems,  therefore  representing  a  constructive  view  of  programs  in  context.  The 
hypothesis  is  that  more  lexical  links  among  programs  may  be  correlated  with  the  overall 
higher  program  total  costs.  The  correlation  between  the  overall  LLA  match  score  and  the 
program  total  cost  found  in  the  weapon  data — which  includes  RDT&E  and  procurement 
costs  together — is  0.21 ,  with  a  p-value  <  0.032.  This  indicates  there  is  a  statistically 
significant  relationship  between  the  number  of  lexical  links  as  an  interdependency  measures 
among  programs  and  total  cost  of  programs. 

Similarly,  a  programmatic  view  of  an  LLA  matrix  can  be  generated  by  using  weapon 
systems  as  columns  and  program  elements  as  rows.  The  correlation  between  the  overall 
LLA  match  scores  and  total  program  costs  is  0.13  with  a  p-value  <0.12.  This  indicates  that 
this  correlation  is  not  statistically  significant  based  on  the  analyzed  data. 

An  operational  view  of  the  LLA  matrix  was  generated  by  using  weapon  systems  as 
columns  and  UJTLs  as  rows.  The  correlation  between  the  overall  LLA  match  scores  and 
total  program  costs  is  0.086,  with  a  p-value  <  0.12,  indicating  that  this  correlation  is  not 
statistically  significant. 

From  an  acquisition  management  and  resource  analysis  perspective,  we  conclude 

that 

■  Major  programs  are  interdependent  on  one  another.  Interdependence  can  be 
shown  by  their  lexical  links  in  budget  documentations  in  constructive, 
programmatic  and  operational  views.  The  degree  that  programs  are 
interdependent  can  be  measured  by  the  number  of  lexical  links. 

■  Highly  interconnected  programs  in  a  constructive  view  are  statistically 
significantly  and  more  expensive  than  less-interconnected  programs  (correlation 
0.21 ,  p-value  <  0.032).  The  word  hubs  selected  from  LLA  suggest  the  “threads” 
that  link  a  portfolio  of  programs  through  shared  resources.  As  an  example,  in 
Figure  18  ADVANCED  MEDIUM  RANGE  AIR-TO-AIR  MISSILE  (AMRAAM)  and 
AIR  INTERCEPT  MISSILE  -  9X  (AIM-9X)  are  connected  through 
“COUNTERMEASURES,”  which  may  share  resources  from  PE  030140N. 
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Figure  17.  A  Constructive  View:  An  LLA  Matrix  Weapon  Systems  versus  Weapon 

Systems 

(Note:  The  correlation  between  the  LLA  overall  match  scores  and  total  program  costs 

is  statistically  significant.) 
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Figure  18.  A  Programmatic  View:  Weapon  Systems  versus  Program 

Elements 
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Figure  19.  An  Operational  View:  Weapon  Systems  versus  UJTLS 


Our  near-term  plan  is  to  apply  the  method  jointly  with  the  unstructured  data  with  the 
MMT  databases  to  illustrate  if  the  LLA  method  can  be  used  to  address  the  following 
questions: 

5.  The  narrative  sections  reference  program-to-program  interdependencies 
(e.g.,  Wideband  Gapfiller  System  flies  on  an  EELV  launch  vehicle).  How 
could  this  be  compared  with  program  interdependence  information  from  the 
DAES,  or  the  ISP  from  our  data  set? 

1 1 .  Are  these  programs  more  or  less  likely  to  incur  cost  growth  relative  to 
their  milestone  B  baselines?  Are  they  more  or  less  likely  to  breach 
their  cost/schedule/performance  baselines? 

12.  How  do  we  determine  the  correlation  using  metrics  that  fundamentally 
affect  acquisition  decision-making?  For  example,  total  program  cost 
and  cost  growth  relative  to  the  Milestone  B  baseline  cost.  (To  do  that, 
we  would  need  to  capture  the  total  program  cost  (development, 
procurement,  and  the  two  combined)  estimated  at  milestone  B,  and 
compare  that  with  these  values  at  milestone  C.  These  data  are  in  the 
MMT  data  set.) 

13.  Can  LLA  of  budget  documentation  provide  an  aggregate  dollar  figure 
that  describes  the  value/magnitude  of  resources  being  shared  among 
these  entities?  Is  this  a  reasonable  proxy  for  the  degree  or 
significance  of  interdependence? 

14.  Is  there  additional  latent  risk  to  programs  that  share  resources?  Is 
there  potential  for  unanticipated  “ripple  effect”  that  could  magnify 
budget  perturbations?  Can  these  effects  be  modeled  or  predicted? 
Would  this  suggest  that  new  approaches  to  budget  analysis  are 
needed? 
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Large-scale  and  Real-time  Consideration 

A  large  number  of  CLA  agents  work  together  in  a  parallel  fashion.  This  allows  the 
LLA  method  to  scale  up  to  distributed,  large-scale  and  real-time  data  sources.  At  the  time  of 
this  printing,  we  have  prototyped  a  multi-agent  network  of  ~10  to  100  agents  in  the  NPS 
High  Performance  Computing  Center  (HPC)  in  the  Hamming  Linux  Cluster  (HLC),  which 
provides  the  requisite  supercomputing  for  the  visualization  of  the  results.  Servers  are  also 
being  built  in  the  NPS  Secure  Technology  Battle  Lab  (STBL)  to  process  classified  data. 

Conclusion 

We  show  in  this  paper  how  to  use  the  Lexical  Link  Analysis  (LLA)  to  match  system 
features  with  those  defined  in  the  original  requirements,  discover  relationships  among 
systems,  and  identify  gaps  with  respect  to  warfighters’  needs.  We  initially  validate  the  LLA 
method  and  show  results  by  correlating  program  interdependencies  resulted  from  the  LLA 
method  with  those  from  subject-matter  experts.  The  Pearson  correlation  for  a  sample  of  461 
program  elements  (PEs)  is  0.39  with  a  p-value  <  0.0000001.  This  indicates  the  positive 
correlation  between  the  LLA  identified  links  as  compared  to  human-analyst-identified  links 
and  that  they  are  reasonably  correlated  with  statistical  significance.  We  also  found  that 
Major  Defense  Acquisition  Programs  (MDAP’s)  are  interdependent  from  one  another 
and  that  such  interdependence  can  be  shown  by  their  lexical  links  in  documentations  in 
constructive,  programmatic,  and  operational  views.  The  number  of  lexical  links  can  be  used 
as  a  metric  to  measure  interdependencies  among  new  technologies.  Highly  interconnected 
programs  in  a  constructive  view  are  statistically  significantly  and  more  expensive  than  the 
less-interconnected  programs  (correlation  0.21,  p-value  <  0.032).  Ultimately,  in  this  vein,  we 
seek  to  use  the  LLA  method  to  automate  and  improve  program  self-awareness  and  make  it 
feasible  for  acquisition  decision-makers  to  analyze  and  dynamically  monitor  large- 
scale  acquisition  documents.  The  resulting  system  analyses  will  facilitate  real-time  program 
awareness  and  can  reduce  the  workload  of  decision-makers  who  would  otherwise  perform 
the  relations-building  task  manually,  thus  making  a  profound  impact  on  the  agility  and 
perhaps  the  long-term  success  of  acquisition  strategies. 
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■  R-TOC  AEGIS  Microwave  Power  Tubes 

■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 

Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 

■  Knowledge,  Responsibilities  and  Decision  Rights  in  MDAPs 

■  KVA  Applied  to  AEGIS  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 
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www.acquisitionresearch.org 


Program-Awareness  via  Lexical  Link 

Analysis  (LLA) 

Dr.  Ying  Zhao,  Dr.  Shelley  P.  Gallup,  Dr.  Douglas  J.  MacKinnon 

Research  Associate  Professors,  Distributed  Information  Systems 
Experimentation,  Naval  Postgraduate  School 


Research  Questions 


Conceptual:  1)  Can  the  information  that  emerges  from  the  acquisition  process  be 
used  to  produce  overall  awareness  of  the  fit  between  programs/projects/systems 
and  needs  for  which  they  were  intended?.  2)  If  a  higher  level  of  awareness  is 
possible,  will  that  enable  system  level  regulation  of  programs/projects/sy stems 
for  improvement  of  the  acquisition  system? 

Focused:  1)  Based  on  the  normal  evolution  of  documentation  and  current  data- 
based  program  information,  can  requirements  (needs)  be  connected  to  system 
capabilities?  2)  Can  requirements  gaps  be  revealed? 


Theory  development:  Is  there  a  correlation  between  system  interdependency 
(links/relationships)  and  development  costs? 


Methodology:  Is  it  possible  to  use  natural  language  and  other  documentation 
(roughly,  unformatted  data)  to  produce  visualization  of  the  internal  constructs 
useful  for  management,  through  lexical  link  analysis  (LLA)? 
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Critical  Need:  Automation 


*ZTL* 

2  g 

D15E 


DOTMLPF 

Analysis 

CBA 


1 

1  ^ 

OCR 

CDD 

CRD 

MSA 


A 


Technology 

Development 


Prod  & 
Deployment 


MOD 


MSA 


MSB 


MS  C 


□ 


Spons  or  Activity 


=  JCIDS  Document 


OA 


=  Acquisition  decision 


JCIDS  Process  and  Acquisition  Decisions 
(From  J-8  CJCSI  3170.01G)(JCIDS,  2009) 


Multiple  Portfolio  Views: 

•  Systems  vs.  Capabilities 

•  Investment  vs.  Capabilities 

•  System  Context 

•  Highly  dependent  programs 
(Joint  Enablers) 

•  Procurement  Optimization 

•  S&T  vs.  future  needs 

•  Sustainment  Efficiency 

•  Market  Value 


•Data  is  too  voluminous,  unformatted 
and  unstructured! 

•Need  automation 

•Extract  relations  among  PE, 
MDAP  and  ACATII 


NiLVdl  Jd  LJJlIi:  VfefciiOl 

Mtinlcrcy,  <Li\ 
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•Extract  costs 


LLA  for  Analysis  of  Unstructured  Data 


DtSE 


•  Apply  Collaborative  Learning  Agents 

-  Separate  patterns  and  anomalies 

-  Parallel  computing  using  NPS  High  Performance  Center  (HPC) 

•  Develop  Visualization 

-  AutoMap 

-  Radar 

-  Matrix 


•  Conduct  Pre-processing  Steps 


-  Named  Entity  Extraction 

•  Leave  out  people,  places  and  organizations 

-  Parts  of  Speech  Tagging 

•  Separate  nouns,  verbs,  adjectives,  adverbs 


Mil -kill  Llll  LlJlltT  VhllOl 

Mtmlcrcy,  CA 
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Apply  Learning  Agents  to  Perform  LLA 


£  si 

D15E 


Agent  collaboration:  multiple  agents  work 
together  for  anomaly  detection 


Anomaly  Search 
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CA 


What  is  a  learning  agent? 


DISE 


•  A  computer  program  or  software 

-  Installed  in  a  computer  with  permission 

-  Perform  automatic  tasks 


•  Multi-agent,  distributed  networks  are  capable  of 

-  Self-managing  (Hinchey  et  al,  2006) 

-  Self-healing  (Dashofy  et  al,  2002) 

-  Self-optimizing,  self-configuring,  self-adapting... 


•  Our  learning  agent 


-  Related  to 

•  Reinforcement  learning  (Sutton  1998) 

•  Bayesian  belief  networks  (Pearl,  1986;  Ben-Gal,  2007) 

•  Hidden  Markov  Models  (Huang  1990) 

-  Learning  patterns  and  anomalies 


Mil -kill  Llll  LlJlltr  VhilOl 

Mtmlcrcy,  <LA 
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Example 

Analysis  of  Urgent  Need  Statements  (UNS) 


*  2  * 

D15E 


•  Analyzed  three  lists  of  classified  needs 
statements  and  links  to  Trident  Warrior  10 
technology  capabilities 

-  Navy  classified  UNS 

-  C5F  (5th  Fleet) 

-  Integrated  Priority  List  (IPL) 

•  CENTCOM  and  NAVCENT 


•  Validity  checked  by  Subject  Matter  Experts 


Mil -kill  Llll  LlJlltr  VhilOl 
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Visualization 
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Lexical  Links  between  System  1  and  2 


Unique  Features  of 
System  1 


Unique  Features  of 
System  2 


Gap  categories 


EHF-TAOfiMC 
Ift-OOUBTOftlSHM 

OMEPiDYivar 


Categories  found  automatically 


I i— i--  w  "TtfE^JI 


m^NI^Gr. 


HF  If^-QFSWTOte 
CCMPUTER-l^iVORKS 
FREQUE^'-RANiES^SJ^UM 
”  SERVER-MAPS 


POR-STAiU- 


CHANGE 


NireNET-^IMmECTMTIYSPRNF 


E\m-wtaA3CE^ 


CEVTTER-OTra^ 


READtfVJESS ^rnm 


sa^mmGpmiw 


QDmOtHXM&JtX^ 


cd(vr™&-5^^ 


WHIJIE-OlWAJMCfiJOCi 


sapR-ssMc^sa 


DMIOT^DSEKM^UTY 


42  of  67  (62%)  of  U NS 
are  matched  in  TW10 


mmim&BM 

CAPABILITY -D&0DF » -mOOF/IVESQ 


IWTEROFKASLP'^^liril^OOAJ^ 


0BFENS&MS3MR 


Un-matched  UNS 
Marched  TW10 

9 


GBQADCASMATOniL/ 
EDSESB^'XI-rjmA^UCnJ^CFN" 

,wi 

BFT-owi^jyi-r 
OE-CPqSHORE 

TOVEFOR^^RmJH^O'.RAOD 


■»cop-mflFoyxMr 


JTRS4^RRA0K» 


E0[>4A£lhE£KV/fr 
PAftTN|IHWT7B>i 
aj^-rx;cap/sut‘- 
VC^JV^CW«I> 
KU-GB«CAP*BUTY 


CQT.flVti  iWfJlp  w 

SI  ViEPiCF 

NMfTM-I^RCOOIW 

so-ro/yDR^cra 

N^Nibc^rr-T^H 


Overlapping  categorie 


ffiKNS  X 
MaCtfA0Lr?f^ 


MF  «- 


K)^B\CPERCT04S 
L  RE0UIR9yCNT- 
|\  SHIFMFBWK^IS 

AKHW^h^SEAFCRI 
K  NETWORK- 


Large  Scale  Data 


DISE 


•  Program  Elements 

•  Programs 

-  Major  DOD  Acquisition  Programs  (MDAP) 

-  Acquisition  Category  II  (ACAT  II) 

•  UJTLS 


Source 

-  Rob  Flowe  OUSD(AT&L)/ARA/EI 
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DOD  Program  Elements 

http://comptroller.defense.gov/defbudgetyfv2009/index.html 


DISE 


Air  Force  Financial 
Management  &  Comptroller 


HOME  ORGANIZATIONS  LEADERSHIP  BUDGET  NEWS  PHOTOS  ART  QUESTIONS 


U.S.  AIR  FORCE 


JOIN  THE  AIR  FORCE 


Budget  >  Air  Force  President's  Budget,  FY08 

FY  2008  Air  Force  Budget  Materials 

The  Air  Force  is  engaged  24/7/365  days  a  year  in  National  and  Worldwide  response  operations.  The  Air  Force  is  focused  on 
continual  refinement  of  performance  measures  and  is  committed  to  providing  good  stewardship  of  resources  which  are 
entrusted  to  them. 


%  Supporting  Documents 


Inside  SAF/FM 


A  FY08  Budget  Rollout  Brief 

U  FY08  Performance  Based  Budgeting  Overview  Book 


It  Military  Construction 


A  Air  Force  MILCON,  FY08 
A  Air  Force  Reserve  MILCON,  FY08 
A  Air  Force  Reserve  MILCON,  FY09 
A  Air  National  Guard  MILCON  FY08 
A  Air  National  Guard  MILCON,  FY09 
Ai  Military  Family  Housing,  FY08 


It  Military  Personnel  Programs 


A  Air  Force  MILPERS,  FY08 
A  Air  National  Guard  MILPERS,  FYI 
A  Air  Force  Reserve  MILPERS,  FY08 


It  Operations  &  Maintenance 


A  Air  Force  O&M,  Vol  1,  FY08 
A  Air  Force  O&M,  Vol  2,  FY08 
A  Air  National  Guard  O&M,  Vol  1,  FY08 
A  Air  National  Guard  O&M,  Vol  2,  FY08 
A  Air  National  Guard  O&M,  Accompanying  Exhibits,  FY08 
A  Air  Force  Reserve  O&M,  FY08 


It  Global  War  on  Terror 


A  Air  Force  MILCON  GWOT  Vol  I,  FY08 
A  Air  Force  MILCON  GWOT  Vol  II,  FY08 


A  Aircraft  Procurment,  Vol  1,  FY08 
A  Aircraft  Procurement,  Vol  2,  FY08 
A  Ammunition  Procurement 
,  At  Missile  Procurement,  FY08 
,  Al  Other  Procurement,  FY08 


It  Research,  Development  Test  &  Evaluation 


The  RDT&E  volumes  are  cross  bookmarked  using  a 
specific  filename  structure.  In  order  for  the  bookmarks  to 
function,  download  each  file  to  the  same  directory  on  your 
local  machine.  The  files  should  be  saved  as: 

AF 3 6 0 0_RDTE_FY  0  8_P  B_v  1  pdf 
AF3600_RDTE_FY08_PB_v2.pdf 
AF3600_RDTE_FY08_PB_v3.pdf 

f[The  RDT&E  volumes  can  be  viewed  directly  via  this 
ebsite,  but  the  inter-volume  bookmarks  will  not  function. 

RDT&E,  Vol  1,  FY08 


r".  ■  "T"'  1 


If  Base  Realignment  and  Closure  (BRAC) 


A  BRAC  1995  Commission,  FY08 
A  BRAC  2005  Commission,  FY08 


If  Working  Capital  Fund 


A  Working  Capital  Fund,  FY08 


Q,  search  FM  I^Xilll 

Advanced  Search 
All  RSSQ 


’ll  La  ks  to  Budget  Materials 


ie-wide  Budget  Documentation 
judget  Documentation 
I  fidget  Documentation 
i  fidget  Documentation 


If  Pr  vious  Year's  Budget  Materials 


Curre  ft  Fiscal  Year 


•  Fiscal 


tear  2010 


Fiscal  fear  2009 


•  Fiscal 

•  Fiscal 

•  Fiscal 

•  Fiscal 

•  Fiscal 

•  Fiscal 

•  Fiscal 


tear  2008 
fear  2007 
|ear  2006 
tear  2005 
lear  2004 
tear  2003 
tear  2002 

^niear  2001 

Insfcalrear  2000 
Fiscal  Year  1999 
Fiscal  Year  1998 
Fiscal  Year  1997 


Trusted  sites 


zl 

\  100%  -  ^ 
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PE  Narrative  Justification 
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DiSE 


PE  NUMBER:  0903421 F 

PE  TITLE:  GLOBAL  POS  HOMING  SYSTEM 


04  Advanced  Conii? 

Cost 


Total  Program 

^-993  GPS  m 

{U)  A.  Mission  Description  and  Budget 


Exhibit  R-2,  RDT&E  Budget  r 


Narrative 
Justification 


— 

|0603421F  GLOBAL  POSITIONING  SYSTEM 

FY  2009 

FY  2010 

FY  2011 

FY  201 2 

FY  2013 

Cost  to 

T  ota 

Estimate 

Estimate 

Estimate 

Estimate 

Estimate 

Complete 

S6S.S  52 

339. S6S 

755.699 

642.740 

569.  SS5 

Continuing 

|3D 

S6E.S52 

S39.S6S 

7 55.699 

642.740 

569.  S35 

Contmiiina 

I3D 

m 

CU) 

to 

CU) 

(U) 


CU) 


Xavstar  Global  Positioning;  System  CGPS)  is  a  space- based  radio  posiioning;  navigation.  and  time  (FNT)  distribution  system.  Tins  Program  Element  {FE>  fluids  die 
Research  and  Development  (RLvD)  for  GPS  D  space  vehicles  (SV)  and  die  nest  generation  Control  Segment  (OCX).  This  includes,  but  is  not  limited  to:  advanced 
concept  development,  systems  engineering  and  analysis,  s  a  cell  ice  systems  development,  die  study  of  augmentation  systems,  modernized  control  segment 
development,  user  equipment  interfaces,  training  simulators.  Integrated  Logistics  Support  {ELS}  products,  and  developmental  test  resources. 

Funds  will  support  engineering  smdies  and  analyses,  architectural  engineering  studies,  trade  suidies:  systems  engineering;  system  development;  test  and  evaluation 
efforts,  and  mission  operations  in  support  of  upgrades  and  product  improvements  for  military  and  civil  applications  necessary  to  support  efforts  to  protect  US. 
military  and  a  hies'  use  of  GPS.  Additionally,  funds  will  ensure  a  disciplined  Capability’  Insertion  Program  plan  to  meec  Joint  Requirements  Oversight  Council 
(JROC)  approved  required  capabilities.  Funds  wfli  support  science  and  technology',  technology7  development  and  systems  development  to  meet  a  Block  approach 
(i.e..  Block  EQ  A:  Block  III  B„  etc.j. 

In  the  FY07  FB,  a  re s mu c cure  of  the  GPS  HI  program  provided  funds  for  the  GPS  III  SV  and  OCX.  The  FYOE  PB  completes  the  GPS  IH  restructure.  Funding  for 
OCX  supports  an  additional  Prime  Contractor  to  support  OC  X  concept  development;  which  includes^  m  addition  to  GPS  EQ  capabiSides.  the  ability  to  control 
modernized  signals. 

This  program  is  Budget  Activity  4  -  Advanced  Component  Development  and  Prototypes  because  it  is  in  Phase  A  {Concept  Development}. 

B.  Program  Change  Summary  fS  in  Millions  ) 


F  Y  2006 


FY  2G07 


F  Y  2003 


FY  2  0: 1 


Previous  President's  Budget 

C  urrent  PBR  President's  Budget 

Total  Adjustments 

Congressional  Program  F.edu cions 

Congressional  Rescissions 

Congressional  Increases 

Reprogrammings 

SBIR  STTR  Transfer 

Significant  Frograin  Chnnypv 

FY06\  — S6.999  for  GPS  EE  development  efforts 


£5. 172 

315.314 

492.094 

731.6' 

£9.556 

313.401 

587.226 

S6S.S: 

4.3  S4 

-1.917 

-1.194 

-0.723 

6.999 

-2.615 

0.004 

R-1  Line  hem  No.  32 

. 633 . 

UNCLASSIFIED 
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Acquisition  Documents 
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PDF  Version  of  Approved 
Universal  Joint  Task  List  (UJTL)  Database 
With  Conditions 

This  update  contains  UJTL  Tasks  approved  In  the  following 
Joint  Staff  Action  Package: 

JSAP  J-7A  00202-08  [Renumbered  to  J-7A  30017-09) 

27  CNO/PSYOP  UJTL  Task  Changes 

DJS  approval  date:  20  February  2009 


Version  3  -  Posted  12  March  2009 

JOINT  STAFF 
WASHINGTON,  D.C.  203 IS 


PROGRAM  ACQUISITION 
COSTS  BY 
WEAPON  SYSTEM 


Department  of  Defense  Budget 
For  Fiscal  Year  2008 

February  2007 


Program  Elements:  RDTE 

books 

-  Air  Force 

-  Army 

•  http://asafm.armv.mil/Docu 
ment.aspx?OfficeCode=1 20 

0 

-  Navy 

•  http://www.finance.hq.navv. 

mil/fmb/1 1  pres/BOOKS.htm 

Universal  Joint  Task 
List 

Weapon  Books 


Llll  LlJlltr  Kl'kMlOl 
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Critical  Need:  Automation 
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Spons  or  Activity 


=  JCIDS  Document 


OA 


=  Acquisition  decision 


JCIDS  Process  and  Acquisition  Decisions 
(From  J-8  CJCSI  3170.01G)(JCIDS,  2009) 


Multiple  Portfolio  Views: 

•  Systems  vs.  Capabilities 

•  Investment  vs.  Capabilities 

•  System  Context 

•  Highly  dependent  programs 
(Joint  Enablers) 

•  Procurement  Optimization 

•  S&T  vs.  future  needs 

•  Sustainment  Efficiency 

•  Market  Value 


•Data  is  too  voluminous,  unformatted 
and  unstructured! 

•Need  automation 

•Extract  relations  among  PE, 
MDAP  and  ACATII 
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•Extract  costs 


LLA  Methodology  Can  Help! 


fu 
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DISE 
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LLA  automates  the  possibility  to 
develop  awareness  of  the  “fit” 
between  PE’s,  budget  and 
warfighter  requirements. 


RDTE  Program  Elements 
(DOD  Budget  $$$ 
Justification) 


How  to  validate  LLA? 


Do  PEs  or  Programs 


match  requirements? 


•  Do  inter-connected  PEs  or  Programs  cost  more? 

T 


Weapon  Book 

(Final  Products  for 
Procurement) 
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PE  Links  Identified  by  Human  Analysts 
(Used  for  LLA  Validation) 
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Exhibit  R-2a,  RDTSE  Project  Justifies 


BUDGET  ACTIVITY 

05  System  Develo-pinent  and  Demonstration  (SDDji 


PE  3  TIT  _E 

0&04i&T2F  Armament'OriJnance 
De  yg-Lopm  ent 


I 


May  2QQS 


PROJECT  HUUfl ER  AND  T  ITLE 
5  361  Stores- Aircraft  I nEerf a ce 


Cost  (J-  in  Millions) 

FY  2003 

FY  2005 

FY  201 0 

'"■m 

FYr  2012 

FY  2013 

FY  20-14 

FYr  2015 

Cost  to 

Taial 

Acrctal 

Estimate 

Estimate 

l  ri  : 

Esn  mate 

E  stimaie 

Estimate 

Estnrate 

C'omnlete 

5361 

Biares-Aircraf:  Interface 

0 .OOO 

0.000 

45.6S5 

. 

0.000 

0.000 

0  005 

0.000 

C  Dnrmring 

TBD 

Quantity  of  ELDXifcE  Articles 

O 

0 

0 

0 

O 

0 

0 

In  l  201 0.  Project  53-151  _  Scores -Aircraft  Interface  (a™),  efforts,  weie  — ansferred  fro  no.  PE  060501  IF.  FJ>i  Si'E  for  Appng  Aircraft,  Project  65-mSSn,  Universal  Am.mep; 
Interface  (UAXL  in  older  to  properLy  fund  rhe  mamting  tec hnalogy 
£U)  A.  MjjMg  Dgsaffitiw  find.  Bgdgd  Item  Jjj^ratiga 

Universal  Anonent  Interface  (UAJ)  is  an  Air  Force  untiauve  to  deveLop.  en nance,  and  implement  standardised  interfaces  lo  aircraft,  weapons  and  mission  pLanmng  to 
support  inc3ZTari.cn.  of  weapons  independent  of  aircraft  Operation  Flight  Program  £OFP>  cycles.  UAI  Ls  currently  being  implemented  on  the  F-I5E  and  F-ld  Block 
40  50  atroaS,  Small  Diameter  Bomb  (SDB)  I  and  EL  Joint  Direct  Attack  Mutuioi  (JDAhl).  Joint  A  ur-  do  -  S  urfa «:  e  Stand-off  Miss  Lie  (JASSM)  and  Precision  ■Guided 
Munitions  Plan  m  n.jr  S  oftwaxe  (P^-MPS)  Additional  aircraft  and  weapons  Live  piozran  plans  to  implement  UAJ.  The  UAI  piogian  office  ls  responsible  for 
development  and  enhancement  of  die  s-andard.  provu-Lon  of  certL&canon  cools  (aest  assets)  and  implementation  support  co  adreraf:  and  weapons. 

The  UAI  efforts  were  transferred  (1)  to  ensure  continued  fund 115  for  UAJ  dinnigb  die  FYDP  (PE  0605  01  IF  will  be  zeroed  out  in  FY  2010  due  10  higher  Air  Force 
prionn.es).  and  (2)  to  properly  fund  rhe  maturing  technalo  gy.  The  new  project  number  ls  esmblisbed  eo  provide  greater  visibility  into  T_: Al's  budges.  Funding  UAI  via 
die  Arm  Ord  PE  will  ensure  fhas  platform  and  weapon  program  ofifrees  have  the  support  required  to  implement  and  update  UAI 

This  program,  ls  in  Budges  Actistty  5  -  System  Develop  men:  and  Demonstration  (&DD)  because  Lt  supports  armament  integration,  an  EDO-type  activity. 
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Exhibit  R-2.  RDT&E  Budget  Item  Justification:  PB  2011  Air  Force 

DATE:  February  2010 

APPROPRIATION/BUDGET  ACTIVITY 

3600:  Research.  Development  Test  £  Evaluation,  Air  Force 

BA  7:  Operational  Systems  Development 

R-1  ITEM  NOMENCLATURE 

PE  0101 1 1 3F:  B-52  SQUADRONS 

000  2698 .5  7;  OWN  ERS 


000269  7.5  2;  OWNERS 


0002697.32;  OWN  ERS 


000  259  6 . 3  6 ;  E  XTR  E  M  E 
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The  B-52  Extremely  High  Frequency  (EHF)  will  integrate  and  install  the  B-52  fleet  with  assured  and  survivable  two-way  EHF  SATCOM  link  for  Emergency  Action 
Messages  (EAMs)  and  report-backs  to  meet  Joint  Chiefs  of  Staff  (JC5)  nuclear  protected  Information  Exchange  Requirements  (IER).  The  B-52  EHF  will  integrate  the 
riiirnty  ut  h.  ivanced  Beyond-Line-of-Sight  {BLOS)  Terminal  {FAB-T)  Increment  1  system  developed  and  procured  by  Space  and  Missile  Command  {SMC)  through  PE 
0303E01F  The  FAB-T  system  consists  of  the  Operator  Interface  Group,  Modem  Processor  Group,  and  Antenna  Group.  The  B-52  EHF  will  integrate  the  following 
capability  into  the  CQNECT  baseline  B-52  architecture:  a  high  data  rate  BLGS  communication  link  supporting  IP-based  Global  Information  Grid  (GIG)  interoperability. 
The  two  Multi-function  Color  Displays  (MFCDs)  and  the  additional  J -Series  Messages  that  were  to  be  integrated  into  CONECT  with  the  B-52  EHF  have  been  moved  to 
the  Strategic  Radar  Replacement  and  CONECT  programs,  respectfully.  In  addition,  the  automated  reporting  of  aircraft  fuel  level  status  off-board  the  jet  capability  will 
also  be  moved  from  the  B-52  EHF.  Disposition  of  this  capability  is  pending  an  approved  acquisition  strategy.  The  B-52  EHF  program  is  planned  to  be  accomplished  in 
three  increments.  Increment  1  is  the  up  front  program  planning  and  risk  reduction  trade  studies  on  items  like  radome  mounting,  environmental  cooling  system  (ECS) 
capabilities,  antenna  boresighting,  etc.  Increment  2  will  integrate,  and  install  the  FAB-T  equipment  for  strategic  connectivity,  as  well  as  implement  trade  study  solutions 
In  addition,  the  ECS  will  need  to  be  upgraded  or  replaced.  The  ECS  modification  requirements  will  allow  enough  margin  to  accommodate  near-term,  -future  roadmap 
efforts.  Finally,  Increments  will  provide  GIG  and  net  ready  capability  as  well  as  full  integration  with  other  B-52  systems. 

Trainers  and  upgrades  for  CONECT  S  EHF 

In  order  to  maintain  currency  with  the  latest  aircraft  configuration,  the  CONECT  and  EHF  programs  will  update  existing  trainers  or  use  computer-based  training  to  add 
CONECT  and  EHF  functionality  to  meet  user-training  requirements  and  establish  a  system  integration  laboratory  (SIL)for  updates  of  the  Weapon  System  Trainers 
(WST). 

Advanced  Targeting  Pod  Functionality 

The  B-52  Modernization  program  fully  integrates  Advanced  Targeting  Pods  (ATP)  by  linking  pod  control,  display  and  target  geo-location  with  the  B-52  Offensive 
Avionics  System  (OAS).  The  B-52  ATP  effort  continues  the  ATP  (Sniper  or  LI  TEN  I NG)  integration  effort  that  began  in  FY07  with  GWOT  funding.  The  ATP  effort 
develops  aircraft  software  updates  to  add  and  incorporate  advanced  pod  functionality  into  the  B-52.  In  addition,  this  effort  upgrades  the  software  1u notions  of  the 
new  Alternate  Mission  Equipment  (AME)  {Mufti  Function  Display  and  the  Integrated  Hand  Controller),  developed  and  procured  under  the  B-52  Advanced  Weapons 
Integration  (A,WI)  modification,  and  enables  the  B-52  to  utilize  a  LI  TEN  IN  G  or  Sniper  pod.  This  effort  provides  hardware  and  software  upgrades  to  the  existing  aircrew/ 
maintenance  trainers  and  the  SIL. 
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Results/Conclusions 
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•  Provided  an  automated  tool  to  surface  important  aspects  among 
programs 

•  Proved  LLA  Validity  for  automation 

-  Adequately  models  expected  human  performance  but  faster 

•  Demonstrated  correlation  among  relations  between  programs 

-  Cost  drivers:  Interrelated  and  Uniqueness 

•  Discovered  statistically  significant  correlations  of  Lexical  links 
between  MDAP  and  ACAII,  and  RDT&E  cost 
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Future  Work 
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•  Extract  lexical  links  for  applications 

-  Continue  to  explore  available  acquisition  data 

•  Extract  the  cost  of  MDAP  programs  (PNO)  from  the  PE  documents 

•  Compare  with  SAR  (Selected  Acquisition  Report)  as  in  the  MDAP 
perspective 

-  Search  for  other  correlations  among  other  program  attributes 

-  Identify  more  dependent  variables 

•  Diversity  metrics 

-  Predict  program  costs  using  this  methodology 

•  Cost  and  cost  growth  relative  to  the  Milestone  B 

•  Cascade  effect  of  program  costs 
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Future  Work 
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•  Determine  methods  to  leverage  the  NPS  HPC  to  analyze  larger  data 
sets 

•  Develop  improved  graphic  illustrations  of  findings 

-  3-D 


-  Dynamic 

•  Provide  an  automatic  LLA  service  for  program  self-awareness 

-  Enterprise  Lexicon  Service 

-  Meta-Data  Registry 

•  Establish  a  complex  system  theory  for  a  cross-domain 

-  Law  of  requisite  variety 

-  Design  Structure  Matrix 
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